Limitations and considerations

The limitations and considerations differ according to the loading method you selected in the endpoint settings.

Limitations and considerations when using the Batch Loading loading method

The following limitations apply when using the Batch Loading loading method:

• The following DDLs are not supported:
  • Drop column
  • Rename column
  • Change column data type

• Capturing changes from source tables without a Primary Key or Unique Index is not supported. If you need to capture changes from such tables, you can add a Primary Key using a transformation. Additionally, Primary Key or Unique Index columns cannot contain NULLs. If you are aware that such columns will be populated with NULL values, define a transformation to change the NULLs to non-nullable values.

• JSON fields are not supported as primary keys.

• The Transactional apply Change Processing Mode is not supported.
• The Allow unlimited LOB size LOB handling option is not supported.

• With the following task settings configuration, the replication task will perform a DELETE operation followed by an INSERT instead of an UPDATE:

  • Change Processing Mode is set to Batch optimized apply.

  • The Apply changes using SQL MERGE option is not selected.

  • UPDATEs error handling for Apply Conflicts is set to: No record found for applying an UPDATE: INSERT the missing target record.

  As Google Cloud BigQuery does not support rollback, in the event that the replication task is unable to insert the updated row, the data will be deleted from the target.

Limitations and considerations when using the Streaming loading method

The following limitations apply when using the Streaming loading method:

• The following DDLs are not supported:
  • Drop column
  • Rename column
  • Change column data type

• Capturing changes from source tables without a Primary Key or Unique Index is not supported. If you need to capture changes from such tables, you can add a Primary Key using a transformation. Additionally, Primary Key or Unique Index columns cannot contain NULLs. If you are aware that such columns will be populated with NULL values, define a transformation to change the NULLs to non-nullable values. If the Control Tables are created with the Streaming loading method (see the second bullet above), Replicate will set each Control Table's Primary Key automatically, even if they already exist on the target.

• JSON fields are not supported as primary keys.

• The Allow unlimited LOB size LOB handling option is not supported.
• The Batch optimized apply Change Processing Mode is not supported.
• Switching from the Streaming loading method to Batch Loading requires you to wait a few minutes before performing DML operations on the source tables. The alternative is to reload all of the tables.
• The number of columns per table cannot exceed 2,000.
• Each record cannot exceed 10 MB.

• By default, data sent via Streaming is immediately available for querying, which might result in high billing costs. If you do not need fully up-to-date query results, you can reduce cost and latency on your queries by setting the "max_staleness" parameter on your table.

  For more information on the "max_staleness" parameter, see Manage table staleness.

• By default, Control Tables on which UPDATE/DELETE operations are performed will be loaded using the Batch Loading method, even if the endpoint is set to use the Streaming method. This is because Control Tables are common to all tasks. However, if you know for certain that all your tasks are using the Streaming loading method, you can set a feature flag that will cause records to be streamed into the Control Tables instead. For more information, contact Qlik Support.
• The Suspended tables (attrep_suspended_tables) Control Table is not supported.
• When a table and is dropped from the target and then created with the same name, it might take few moments before Replicate can open a Stream for that table. This may cause a lag in updating that table. Note that this can also occur when the replication task is configured to drop and create existing tables during Full Load.
• Apply Conflicts error-handling policies are not relevant as the Streaming loading method uses UPSERT for INSERT and UPDATE operations.

• The STATEMENT format of errors in the Apply exceptions (attrep_apply_exceptions) Control Table will be as follows:

  {"streamPosition":STREAM_POSITION,"operation":OPERATION}

  Where the OPERATION values can be INSERT, UPDATE or DELETE.